A Lexical Semantic and Statistical Approach to Lexical Collocation Extraction for Natural Language Generation

Dissertation Abstract comprise words for which neither word can be substituted by a synonym or hyponym. Potential colloca-tions comprising certain adjacent part-of-speech tags are extracted from text. An online thesaurus and a lexical database of word classes are queried for synonyms and hypo nyms, respectively, for each potential collocation. These queries create potential challenger pairs, such as " strong java " and " powerful coffee. " A substitution procedure is then applied to determine if any of these challenging word pairs occur more frequently than the potential colloca-tion. The VERIFY lexical collocation-extraction system has been implemented incorporating these ideas. Results to date have been positive; a system using lexical-semantic knowledge , that is, synonymy and hyp-onymy, within a lexical collocation-extraction system outperforms a system using purely statistical knowledge. To compare system output to human judgments of training data, a training component was also incorporated into VERIFY. This component is able to adapt to new data. Overall system performance, measured in recall and precision scores, 2 improved using this component. To provide a more flexible system given a user's application, a weighting mechanism was used to produce a range of recall and precision scores. These weights can be adjusted to optimize system performance. Through an experimental procedure , VERIFY was trained on a set of human-analyzed data and then independently tested on another set of L exical collocations are frequently occurring word pairs in natural language whose presence are not always predictable by their usage. These collocations are used by native speakers of a language almost without thought, yet they must be learned by nonnative speakers of the language. A native speaker of English might say that he/she drinks " strong coffee, " but a nonnative speaker might say either " powerful coffee " or " sturdy coffee. " Collocations tend to vary among languages and topic domains. Unfortunately , the task of correctly identifying lexical collocations, even by native speakers of the language, has been shown to be difficult. Computer systems that translate natural languages, or machine-translation systems, need to know about lexical collocation information to produce natural-sounding or colloquially proper text. Natural language generation is a component of a machine-translation system that automatically produces natural-sounding text in a particular language given a language-independent meaning as input. This dissertation (Doerr 1994) 1 demonstrates how to automatically locate and extract lexical collocations from machine-readable text for use within a machine-translation system's …